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DESCRIPTION 

LIGHT SOURCE ESTIMATION APPARATUS, LIGHT SOURCE ESTIMATION 
METHOD, IMAGE-PICKUP APPARATUS, AND IMAGE PROCESSING METHOD 

TECHNICAL FIELD 

The present invention relates to a light source estimation 
apparatus, light source estimation method, image-pickup 
apparatus, and image processing method in which, for example, 
image-pickup means having a plurality of different spectral 
sensitivity characteristics uses sensor response values obtained 
when photographing the image of an unspecified arbitrary object 
to estimate the spectral characteristics indicating the color of 
the unknown photographing light source which had been 
irradiating an object. 

BACKGROUND ART 

The light which is incident on the human eye to enable 
vision is a portion of the radiant energy due to illumination 
which has been reflected by an object which is seen and has 
propagated through the air; although the human vision system 
cannot directly measure the characteristics of objects and 
illumination, objects can be identified with a degree of 
reliability even under illumination having unknown chromatic 
characteristics. This property is called color constancy, and 
for example enables a white object surface to be perceived as 
white . 
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On the other hand, in digital still cameras, digital video 
cameras and other electronic image-pickup equipment, scenes are 
picked up as images through the response of a CCD (Charge 
Coupled Device) or other photosensor; however, because in 
general the balance of sensor response among the R, G, B, or 
other color channels is constant, in order to form an image in a 
state in which the appearance is natural in accordance with the 
scene illumination, a correction mechanism is necessary to 
adjust the balance between channels. If the balance is not 
adequately adjusted, to the viewer of the image, places normally 
recognized as achromatic objects will be reproduced as colored 
in the image, or objects will be reproduced with a color 
different from the color remembered, so an unnatural impression 
is imparted; hence balance adjustment is extremely important for 
color reproduction of an image. 

Balance adjustment among channels can- be performed by, for 
example, correction of achromatic colors called white balance in 
the gain adjustment of each channel; by correcting the color 
rendering properties of the light source through linear matrix 
transformation of signals among channels (Patent Reference 1); 
or by matching to the different sensitivity responses of the 
sensors of image-pickup equipment, vision systems and similar. 
However, whichever method is used, the correction mechanism must 
use some means to obtain correction parameters appropriate to 
the scene. For example, the following equations (1) and (2) can 



be used to calculate appropriate gain values for adjustment of 
the white balance of sensors with a RGB three-channel response 
which is linear with respect to the quantity of light, together 
with the spectral sensitivity characteristics of the image- 
pickup system, if the spectral distribution of the light source 
for the photographed scene is known. 



where S is a matrix indicating the sensor sensitivity 
(three channels x number n of wavelength samples) , and I is a 
column vector indicating the spectral distribution of the light 
source (number n of wavelength samples). 



However, for the image-pickup equipment, information 
relating to objects existing in the scene at the time of image 
pickup without calibration or the like and the illuminating 
light sources of the scene are normally unknown; and adjustment 
parameters appropriate to the scene, or the chromatic 
characteristics of the illuminating light source necessary to 
determine those parameters, must be identified from the response 
"results of a dedicated sensor or sensor for image pickup, 
constituting a problem known as the light source estimation 
problem or the color constancy problem. 



= SI 



(1) 




= G w /R w , g G = G w /G w =1.0, g B = G w /B w 



(2) 
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In the field of vision studies, various algorithms and 
calculation models began to be proposed from around 1980, and 
apart from these, techniques based on empirical knowledge have 
been incorporated in conventional color image-pickup equipment, 
the estimation performance of which has advanced through the 
years. Recently, applications to robotics and other artificial 
vision systems have also been anticipated. 

One of the most widely used algorithms extracts the color 
components of the light source from average values of sensor 
response and the projection thereof onto a black body locus, 
based on the assumption that the spatial average over the scene 
of the surface reflectivity of an object is close to gray (Non- 
patent Reference 1, Non-patent Reference 2), and is used in a 
variety of modes, such as simply averaging the sensor response 
among pixels, averaging pixels within the range of a specified 
brightness level, or changing the range or weighting of sampling 
depending on the position in space. There are also a method in 
which color components of the light source are extracted from 
sampling results for pixels with high response values, assuming 
that the area with the highest brightness level corresponds to a 
white surface close to a perfectly diffuse reflecting surface 
(Patent Reference 2), and a method in which an area of high 
brightness level is assumed to be a specular reflecting 
component, and the light source is estimated from the 
distribution of the response values (Non-patent Reference 3). 



Because these methods are based on an assumption about an object 
surface, which should be physically independent of the light 
source, it is known that depending on the scene, the results of 
light source estimation may be greatly affected by the state of 
an object which deviates from the assumptions made. 

There are also a study in which, by assuming a reflection 
model in which an object surface is a diffuse reflecting surface, 
and approximating the spectral characteristics of the light 
source and of the object surface by a linear model of few 
dimensions, reconstruction is attempted through linear 
calculations using a vector space different from that of the 
sensor response (Non-patent Reference 4), and a study in which 
constraining conditions, such as that the spectral reflectivity 
of an object surface must physically be in the range 0 to 1, are 
applied to select a light source with high probability (Non- 
patent Reference 5); however, in generalized image-pickup 
systems with few response channels, these do not independently 
provide sufficient estimation performance. Further, although the 
volume of computations is increased, there has also been 
proposed a method of integrating a plurality of known 
assumptions and probabilistic distributions for the light source, 
object surfaces, image-pickup system and similar, to improve the 
accuracy of statistical estimation (Non-patent Reference 6) . 

In methods which apply reflection models in particular, 
rather than performing an estimate taking as the solution a 



single completely unknown light source, in some methods wide 
prior knowledge is utilized in a method of determination in 
which the most probable light sources are categorized or 
detected from among a number of light sources selected in 
advance as candidates; such methods may be advantageous in that 
calculations are comparatively simple and results can be output 
rapidly. As criteria for judging the reliability of the result, 
errors by restoring the sensor response itself under a fixed 
constraint condition may be used (Non-patent Reference 7); and 
there have been proposals for widely using distribution states 
in the color gamut within the sensor space, to efficiently 
quantify a correlation relationship through comparison with a 
color gamut, adopted in advance as a reference, or a weighted 
distribution (Non-patent Reference 8, Non-patent Reference 9, 
Non-patent Reference 10, Patent Reference 3) . 

Patent Reference 1: Published Japanese Patent Application 
No. 2002-142231 

Patent Reference 2: Published Japanese Patent Application 
No. H9-55948 

Patent Reference 3: Published Japanese Patent Application 
No. H5-191826 

Non-patent Reference 1: G. Buchsbaum, "A Spatial Processor 
Model for Object Color Perception", J. Franklin Inst., 310, 1980 
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Non-patent Reference 2: E.H. Land, "Recent Advances in 
Retinex Theory", Vision Research, 26, 1986 

Non-patent Reference 3: H.C. Lee, "Method for computing 
the scene-illuminant chromaticity from specular highlights", J. 
Opt. Soc. Am. A, Vol. 3, No. 10, 1986 

Non-patent Reference 4: L.T. Maloney & B.A. Wandell, 
"Color Constancy: A method for recovering surface spectral 
reflectance", J. Opt. Soc. Am. A, 1986 

Non-patent Reference 5: D.A. Forsyth, "A Novel Algorithm 
for Color Constancy", Int. J. Comput . Vision, 5, 1990 

Non-patent Reference 6: D.H. Brainard & W.T. Freeman, 
"Bayesian color constancy", J. Opt. Soc. Am. A, Vol. 14, No. 7, 
1997 

Non-patent Reference 7: B. Tao, I. Tastl & N. Katoh, 
"Illumination Detection in Linear Space", Proc. 8th Color 
Imaging Conf., 2000 

Non-patent Reference 8: Hewlett-Packard Company, Hubel et 
al., "White point determination using correlation matrix memory", 
U.S. Patent 6038339 

Non-patent Reference 9: G.D. Finlayson, P.M. Hubel & S. 
Hordley, "Color by correlation", Proc. 5th Color Imaging Conf., 
1997 
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Non-patent Reference 10: S. Tominaga & B.A. Wandell, 
"Natural scene-illuminant estimation using the sensor 
correlation", Proc. IEEE, Vol. 90, No. 1, 2002 

DISCLOSURE OF THE INVENTION 

Generally, in order to use a light source estimation 
algorithm in image processing operations such as white balance 
processing within a digital camera, not only must the processing 
speed be fast, but at the time of implementation it is necessary 
that the costs of memory consumption and similar be low. 

However, even in the cases of those among the above- 
described conventional algorithms which enable comparatively 
fast categorization and detection (Non-patent Reference 8, Non- 
patent Reference 9, Patent Reference 3) , as indicated by the 
conceptual diagram of a conventional method to evaluate the 
reasonableness of test light sources in sensor response space in 
FIG. 10, while there is the possibility of improving the 
estimation accuracy as the number 1 to n of test light sources 
101 set as candidates is increased, because comparative 
evaluations of the image distribution 106 for sensor response 
105 to a picked-up image of an object 103 picked up by the 
image-pickup means 104 with reference distributions (1, 2, n) 
108 stored in storage media 107 corresponding to the respective 
test light sources 101, are performed, by the comparison portion 
109, in an image distribution 106 in sensor space dependent on 
the image-pickup light source 102, with scores of evaluation 
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results (1, 2, n) 110 output, and the test light source 

judged to be most correct based on the score values 110 is 
judged to be the estimated light source O by the judgment 
portion 111; information for reference distributions (1, 2, 
n) 108 such as the color gamut, weighted distribution, target 
values, and so on used as comparison criteria for the correct 
light source must be held in storage media 107 in the same 
quantity as the number 1 to n of test light sources 101 selected 
for use, so there is a tendency for increased use of the ROM or 
other storage media 107, and therefore the problem of a 
combination of advantages and disadvantages with respect to 
accuracy and cost has remained. 

Whereas the above methods create one image distribution for 
one picked up image by a fixed projection, and make the 
distribution become judgment criteria as compared with a 
plurality of reference distributions corresponding to a 
plurality of light sources assumed, in the present invention, a 
plurality of reference distributions generated by projections 
corresponding to a plurality of assumed light sources are 
compared with a single fixed reference distribution to be 
employed as the judgment criterion. 

The present invention was devised in light of the above, 
and has as an object to provide a light source estimation 
apparatus, light source estimation method, image-pickup 
apparatus, and image processing method for estimating the color 



characteristics of an unknown light source of an image-pickup 
scene from the sensor response, in order to improve the 
automatic white balance adjustment and other aspects of color 
reproduction quality of color image-pickup equipment. 

A light source estimation apparatus of this invention 
estimates the correct image-pickup light source by including: 
storage means for storing, for each test light source, 
parameters for projecting sensor response values into an 
evaluation space not dependent on the image-pickup light source 
by performing, for the sensor response values, operations which 
can be colorimetrically approximated from a plurality of 
different known spectral sensitivity characteristics of image- 
pickup means and spectral characteristics of a plurality of test 
light sources assumed in advance; projection conversion means 
for projecting the sensor response values into the evaluation 
space not dependent on the image-pickup light source using 
parameters stored in the storage means; and evaluation means for 
evaluating the correctness of a plurality of test light sources 
based on the image distribution state of sample values of an 
image scene projected by the projection conversion means. 

Hence according to this invention, the following action is 
achieved. 

With respect to a sampled sensor response, through 
operations which can be colorimetrically approximated from the 
known spectral sensitivity characteristics of the image-pickup 
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system and from spectral characteristics of test light sources, 
projection into an evaluation space not dependent on the light 
source is performed, and the reasonableness of each of the test 
light sources is evaluated based on the states of sample values 
widely distributed therein. 

Accordingly, it is necessary only to store, with respect to 
each test light source, a matrix or other parameters for 
projection from the sensor space into the evaluation space, so 
that by providing evaluation criteria in a single evaluation 
space, high estimation accuracy can be obtained with low memory 
consumption. 

Further, a light source estimation method of this invention 
correctly estimates, for sensor response values, the image- 
pickup light source, by performing projection into an evaluation 
space not dependent on the image-pickup light source through 
operations which can be colorimetrically approximated from known 
spectral sensitivity characteristics of image-pickup means and 
from spectral characteristics of assumed test light sources, and 
by evaluating the correctness of a plurality of test light 
sources based on the state of distribution of sampled values of 
the projected scene. 

Hence according to this invention, the following action is 
achieved. 

In order to perform evaluations using a fixed space not 
dependent on the light source, it is sufficient to hold 



information, as comparison criteria for the correct light source, 
only for a single reference distribution space, so that 
evaluation processing is simplified, and consequently the 
problem of increasing costs can be resolved. As a further 
consequence, a greater amount of information (conditions and 
data) for referencing as criteria for the correct light source 
can be provided, so that optimization adjustment to improve 
estimation accuracy is also facilitated. 

Further, an image-pickup apparatus of this invention 
includes: storage means for storing, for each test light source, 
parameters for projecting sensor response values into an 
evaluation space not dependent on the image-pickup light source 
by performing, for the sensor response values, operations which 
can be colorimetrically approximated from a plurality of 
different known spectral sensitivity characteristics of image- 
pickup means and spectral characteristics of a plurality of test 
light sources assumed. . in advance; projection conversion means 
for projecting sensor response values into the evaluation space 
not dependent on the image-pickup light source using parameters 
stored in the storage means; evaluation means for estimating the 
correct image-pickup light source by evaluating the correctness 
of a plurality of test light sources based on the image 
distribution state of sample values of an image scene projected 
by the projection conversion means; light source estimation 
means for estimating the final image-pickup light source to be 
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determined as the estimated light source by conjoining in 
numerical formulas, or by selecting through conditional 
branching, or by combining both of an image-pickup light source 
determined by estimation and a light source determined by an 
estimation method different from the estimation method used; and 
color balance adjustment means which uses spectral 
characteristics or parameters appropriate thereto, as the color 
of the estimated image-pickup light source, in color balance 
processing of the sensor response of the image-pickup means. 

Hence according to this invention, the following action is 
achieved. 

In this image-pickup apparatus, the range of estimation of 
the image-pickup light source can be broadened, and by storing, 
for each test light source, only a matrix or other parameters 
for projection from the sensor space into the evaluation space 
and by providing evaluation criteria in a single evaluation 
space, high estimation accuracy with low memory consumption is 
obtained to be used in color balance processing. 

Further, an image processing method of this invention 
performs projection, for sensor response values, into an 
evaluation space not dependent on the image-pickup light source 
through operations which can be colorimetrically approximated 
from known spectral sensitivity characteristics of image-pickup 
means and from spectral characteristics of assumed test light 
sources; estimates the correct image-pickup light source by 



evaluating the correctness of a plurality of test light sources 
based on the distribution state of sample values of the 
projected scene; estimates the final image-pickup light source 
to be determined as the estimated light source by conjoining in 
numerical formulas, by selecting through conditional branching, 
or by combining both of, an image-pickup light source determined 
by estimation and a light source determined by an estimation 
method different from the estimation method used; and uses the 
spectral characteristics or parameters appropriate thereto, as 
the color of the estimated image-pickup light source, in color 
balance processing of the sensor response of the image-pickup 
means . 

Hence according to this invention, the following action is 
achieved. 

In this image processing method, the range of estimation of . 
the image-pickup light source can be broadened, and by storing, 
for each test light source, only a matrix or other parameters 
for projection from the sensor space to the evaluation space, 
and by providing evaluation criteria in a single evaluation 

space, high estimation accuracy with fast processing is obtained 

i 

to be used in color balance processing. 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a conceptual diagram of a method of evaluating 
the reasonableness of test light sources in an evaluation space 
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not dependent on the light source, which is applied to an 
embodiment of the present invention; 

FIG. 2 is an image processing block diagram within a 
digital still camera; 

FIG. 3 is a flowchart showing reference distribution 
generation processing ; 

FIG. 4 is a flowchart showing light source estimation 
processing; 

FIG. 5 is a diagram showing a spectral reflectivity basis 
function; 

FIG. 6 is a diagram showing an example of projecting a 
color chart into a reflectivity vector space; 

FIGS. 7A to 7C are diagrams showing test light sources, in 
which FIG. 7A is at equal intervals, FIG. 7B is finely divided 
at specific intervals, and FIG. 7C includes a plurality of light 
sources; 

FIG. 8 is a diagram showing a distribution of reflectivity 
samples ; 

FIG. 9 is a diagram showing a reference distribution tablel- 
and, 

FIG. 10 is a conceptual diagram of a conventional method of 
evaluating the reasonableness of test light sources in sensor 
response space. 



BEST MODE FOR CARRYING OUT THE INVENTION 

Hereinafter, embodiments of the present invention are 
explained, referring to the drawings as appropriate. 

FIG. 1 is a conceptual diagram of the method, which is 
applied in this embodiment, of evaluating the reasonableness of 
test light sources in an evaluation space not dependent on the 
light source. 

In FIG. 1, an image of an object 3 is picked up by image- 
pickup means 4 using an image-pickup light source 2. In order to 
enable projection into an evaluation space not dependent on the 
light source, with respect to a sensor response 5 sampled by the 
image-pickup means 4 at this time, colorimetric approximation 
operations are performed in advance using spectral sensitivity 
characteristic of the image-pickup means 4, which is a known 
quantity, and using spectral characteristics of test light 
sources (1 to n) 1, and a matrix (1 to n) 8 corresponding to 
each of the test light sources, which is stored in storage media 
7, is used to perform projection into image distribution 9 of 
the evaluation space by means of a projection conversion portion 
6; then based on the states of sample values distributed widely 
in the image distribution 9, an evaluation score value 12 is 
output by an evaluation portion 10 with respect to the 
reasonableness of each test light source (1 to n) 1 based on a 
reference distribution 11, and a judgment portion 13 then judges 
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the test light source, which is judged to be most nearly correct 
based on the score values 12, to be the estimated light source 0. 

Accordingly, it is sufficient to store in the storage media 
7 only a matrix (1 to n) 8 or other parameters for projection 
from the sensor space to the image distribution 9 in the 
evaluation space with respect to each of the test light sources 
(1 to n) , so that by providing evaluation criteria through a 
single reference distribution 11 in the image distribution 9 in 
the evaluation space, scores (1, 2, n) 12 having a sufficient 

amount of information to enable accurate judgment by the 
judgment portion 13 can be output by the evaluation portion 10, 
with only a small amount of storage space of the storage media 7 
used . 

FIG. 2 is a block diagram showing an image processing 
system within a digital still camera, to which an embodiment of 
this invention is applied. 

A digital still camera is assumed, in which sensor response 
due to the spectral sensitivity characteristics, differing with 
respect to each pixel, in the three channels red, blue, green 
can be obtained as 10-bit digital values proportional to the 
quantity of light, and in the image processing operation unit 
within the apparatus, processing for white balance adjustment is 
performed using appropriate gain values for each channel. 

In order to determine appropriate gain values for white 
balance adjustment based on sensor response obtained by 



subtracting offset components in a black correction portion 22 
from values read by a sensor response readout portion 21, 
subsampling is performed at appropriate position intervals among 
all pixels by a subsampling portion 23. At this time, pixels 
which can be judged to have reached saturation in ranges near 
the minimum and maximum values of the sensor response, are 
excluded. Light source estimation processing described later on 
is performed by a light source estimation processing portion 24 
on those sampled pixels. 

As a result, a gain value corresponding to the estimated 
light source is selected from among the gain values for white 
balance adjustment for each test light source stored in a gain 
determination portion 25 through the calculations indicated 
previously in equations (1) and (2) ; and the selected gain value 
is employed in white balance processing in a white balance gain 
adjustment portion 26. 

Subsequently, through nonlinear grayscale conversion by a 
grayscale conversion portion 27 and 3x3 matrix conversion by a 
color space conversion portion 28, Y, Cb and Cr 
(luminance/color-difference signals) are converted into 8 bits 
each, encoding by an encoding portion 29 which includes image 
compression processing is performed, and the result is written 
by a file writing portion 30 as an electronic file in a memory 
card. 
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In light source estimation in this embodiment, an object 
surface linear reflection model is assumed as indicated by the 
following equation (3) with respect to the sensor response. 



R 



f = 



G 



= SLr 



(3) 



B 



where L is the diagonal matrix (nxn) containing n wavelength 
samples of light source spectral distributions, and r is the 
column vector (number of wavelength samples n) indicating the 
object surface spectral reflectivity. 

This embodiment is explained using the matrix calculation 
of equation (4), which assumes that the spectral reflectivity of 
the object surface can be approximated by a linear combination 
of three basis functions. 



where B is a matrix indicating the basis functions of the 
spectral reflectivity (number of wavelength samples n x basis 
number 3) , bi, b 2 , b 3 are column vectors indicating the basis 
functions of the spectral reflectivity (number of wavelength 
samples n) , w is a column vector containing weighting 
coefficients (basis number 3), Pi, (3 2 , P3 are weighting 
coefficients used to indicate the spectral reflectivity as 
linear sums of the basis functions, and r a is a column vector 



fix 
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(number of wavelength samples n) indicating approximate values 
of the spectral reflectivity. 

If the spectral reflectivity is known, approximate values 
for weighting coefficients of the basis functions can be 
calculated as in the following equation (5) . 



w ■ 



A 



= B t (BB t Y r (5) 



Because in equation (5) w does not depend on the sensor or 
the image-pickup light source, the vector space resulting from 
the weighting coefficients of basis functions (hereafter called 
the reflectivity vector space) can be said to be a space 
specific to the object. The spectral reflectivity basis 
functions shown in FIG. 5 are examples of basis functions shown 
in the wavelength range from 400 rim to 700 nm; because (31 
represents the brightness component, the first component is set 
to be flat over the wavelength range, whereas the second and 
third components are the results of extracting the highest two 
components excluding the first component as an offset from the 
spectral reflectivity data for 24 specific color charts, and 
then performing principal component analysis. From equations (3) 
and (4), when the basis number and the number of sensor response 
channels are the same, the column vector projected from the 
sensor response by the matrix is calculated using the following 
equation ( 6) . 
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The matrix M in equation (6) projects the sensor response 
into reflectivity vector space, but is a matrix which depends on 
the light source L, and is called hereinafter the light source 
matrix. If the same light source as that for the scene for which 
the sensor response was obtained is used as the light source L, 
the column vector w projected from the sensor response is 
restored to a close value even if the spectral reflectivity of 
the object is unknown. However, if a light source different from 
that for the image-pickup scene is used, this restoration 
accuracy is not obtained. Hence an arbitrary light source Li is 
assumed., and the light source matrix M± indicated in equation (7) 
is used. 

Mi = (SLiBT 1 (7) 

Using the light source matrix Mi obtained using equation (7), 
and utilizing the relation between the column vector w i projected 
as equation (6) and the correct column vector w for the object, 
the degree of similarity with the image-pickup light source in 
the reflectivity vector space can be evaluated. FIG. 6 shows, by 
simulation results for 24 color charts in the P2-P3 plane in the 
reflectivity vector space, that the distribution of points 
approximated from the spectral reflectivity of a known surface 

is in a state close to the distribution of points projected 
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using the light source matrix for the same light source as that 
at the time of image pick up from the sensor response for a 
pickup image of the same surface, and that the distribution of 
points projected using a light source matrix for a light source 
different from that at the time of image pickup is in a state 
different from the former distribution . 

The column vectors w which can be adopted by the object are 
widely distributed in reflectivity vector space, and it is 
difficult to evaluate the relation with an unknown object by 
using the column vector w i obtained from the sensor response for 

a single pixel. Hence here it is assumed that the image-pickup 
scene is illuminated uniformly by a single light source, the 
sensor response for sampled pixels among all pixels is. projected 
into the reflectivity vector space, and by evaluating these 
distribution states (hereafter called the image distribution) , a 
single estimated light source is determined. A plurality of 
light sources for evaluation (hereafter called test light 
sources) are provided, and all light source matrixes are 
calculated and stored in advance according to the above equation 
(7) . 

At the time of estimation processing, each light source 
matrix is employed with respect to all test light sources to 
evaluate the projected image distribution, and the test light 
source with the highest evaluation index indicating the degree 
of correctness is selected as the estimated light source among 



all the test light sources. Here, natural light with color 
temperatures in the range approximately 2850K to 8000K are 
assumed as test light sources; in order to reduce estimation 
error scattering, seven test light sources were adopted on the 
CIE daylight locus in the u' -v' plane of the CIE 1976 UCS 
chromaticity diagram at intervals as nearly equal as possible, 
as shown in FIG. 7A. In this embodiment, any of the test light 
sources can be selected as the estimation result, so that as 
shown in FIG. 7B, the test light sources are more finely divided 
only in specific intended intervals in the color temperature 
direction on the u' -v' plane; and as shown in FIG. 7C, light 
sources employing physically different light emission methods, 
such as fluorescent lamps, may also be added to raise the 
probability of obtaining the correct estimation result in more 
diverse scenes. 

In order to evaluate in relative terms whether the 
distribution state of an image distribution is the correct state 
for an object, a single fixed distribution state (hereinafter 
called the reference distribution) to be taken as the criterion 
for comparison in reflectivity vector space is referenced. The 
reference distribution is stored as data in the format of a two- 
dimensional numerical table provided with weighting coefficients 
with respect to each cell divided at equal intervals in the P2-P3 
plane. This reference distribution is generated using, for 
example, a procedure such as follows. 



Specific reference distribution generation processing is 
shown in the flowchart of FIG. 3. 

In step SI, spectral reflectivity data for the surfaces of 

numerous objects which might be objects for image pickup is 

collected, and representative samples which are as diverse as 
possible are extracted. 

In step S2, sample data for spectral reflectivity is 
projected into reflectivity vector space using equation (5) (FIG. 
8) . 

In step S3, the cell region is set by specifying lower ends 
low 2 and low 3 , upper ends high 2 and high 3 , and cell division 
numbers bin 2 and bin 3 for each axis of the P2-P3 plane in 
reflectivity vector space, such that the rectangular region 
comprehends the sample distribution. 

In step S4, the numbers of data samples positioned in each 
cell region are counted to generate a frequency distribution. 

Cell coordinates (x, y) are calculated using the following 
equation ( 8 ) . 

x = floor ( (P2-I0W2) xbin 2 / (high 2 -low 2 ) ) 

y = floor ( (P3-I0W3) xbin 3 / (high 3 -low 3 ) ) (8) 

where floor ( ) is an operator which discards the decimal 
fraction . 
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In step S5, values Tr xy which encode frequencies in each 
cell to an appropriate bit depth are recorded. 

In step S6, in order to form the outline of the 
distribution range of the reference distribution, the polygon 
which convexly encompasses the cells in which values exist, is 
calculated, and by assigning a value of 1 to any cells existing 
within the polygon for which a value does not exist, holes in 
cells within the outline are filled. FIG. 9 shows numerical 
values in cells in an example of a reference distribution 
generated with a bit depth of 2 . 

FIG . 4 is a flowchart of specific light source estimation 
processing. 

At the time of estimation processing, calculation of score 
values with respect to each test light source is repeated 
according to the following procedure shown in FIG. 4. 

In step Sll, a projection matrix is selected for a test 
light source. Specifically, the projection conversion portion 6 
shown in FIG. 1 selects a light source matrix for a test light 
source i from the storage media 7. 

In step S12, reading of sample pixels is performed. 
Specifically, the projection conversion portion 6 shown- in FIG. 
1 reads sample pixels from the image-pickup means 4. Here, the 
sensor response 5 of sample pixels is the image-pickup result 
for various scenes. 
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In step S13, matrix conversion is performed. Specifically, 
the projection conversion portion 6 shown in FIG. 1 uses the 
light source matrix M ± for the test light source i to project the 
sensor response 5 for sample pixels into the reflectivity vector 
space . 

In step S14, the image distribution is generated. 
Specif ically, the projection conversion portion 6 shown in. FIG. 
1 creates an image distribution 9 in the same cell positions as 
the reference distribution 11. Similarly to generation of the 
reference distribution 11, the image distribution 9 includes 
values Th iX y which encode the frequency in each cell to an 
appropriate bit depth. Here the bit depth is taken to be 1; an 
example of an image distribution in which a value of 1 is 
assigned to each cell in which one or more pixel exists, and 0 
is used for all other cells, is shown in gray within the cells 
of the reference distribution table in FIG. 9. 

In step S15, processing is repeated for all sample pixels, 
returning to step S12 to repeat the processing and judgment of 
steps S12 through S15. Specifically, the projection conversion 
portion 6 of FIG. 1 not only records the image distribution 9 
for each pixel, but also counts pixels positioned in cells 
(shown as the bold border in FIG. 9) for which values exist in 
the reference distribution 11. 

In step S16, score values are calculated with respect to 
each test light source. Here, a score value 12 is a correlation 



value or similar between the image distribution 9 and the 
reference distribution 11. Specifically, the evaluation portion 
10 shown in FIG. 1 calculates the following three types of 
indexes . 

First, as the distribution correlation, equation (9) is 
used to calculate the sum of the product of the image 
distribution 9 with weightings of the reference distribution 11 
for each cell, as an index representing the correlation function 
between the reference distribution 11 and the image distribution 
9. 

bin2 bin3 

^=IZ^\ (9) 

Second, as the division of the number of pixels, equation 
(10) is used to calculate the fraction of the number of pixels 
among all sample pixels existing in the color gamut (shown as 
the bold border in FIG. 9) of the reference distribution 11, as 
a comparative index relative to the reference distribution 11 
with respect to the number of pixel samples. 

Ipi = (number of pixels positioned at cell coordinates x,y 
for which Tr ixy >0) / (total number of sample pixels) (10) 

Third, as the distribution size, equation (11) is used to 

calculate the following index, only with respect to the image 

distribution in reflectivity vector space, based on the 

assumption that in a case of projection using an erroneous light 

source matrix, the larger the difference with the correct light 
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source matrix, the broader will be the distribution range 
dispersed in the (5 2 axis direction. 

Ir± = (Max2 m -Min2 m ) / (Max2i-Min2i) (11) 

where Max2j. is the maximum value of (3 2 projected by the 
light source matrix of the test light source i, Min2i is the 
minimum value of p 2 projected by the light source matrix of the 
test light source i, and m is the light source i among all test 
light sources for which Max2i~Min2i is smallest . 

Here, the evaluation portion 10 shown in FIG. 1 uses 
equation (12) to obtain score values 12 for light sources i by 
multiplying the three types of index. 

Si = ICi-Ipi-Iri (12) 

In step S17, processing is repeated with respect to all 
test light sources by returning to step Sll to repeat the 
processing and judgment of steps Sll through S17. 

In step S18, the estimated light source is selected. 
Specifically, the evaluation portion 10 shown in FIG. 1 
determines as the estimated light source the light source i 
having the highest score value 12 upon obtaining score values 12 
for all test light sources 1. 

In addition, an intermediate light source, selected through 
weighted averaging process etc. using test light sources with 
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high score values, may also be determined to be the estimated 
light source. 

In addition, only a specific interval close to test light 
sources with high score values on the u' -v' plane in FIG. 7A may 
be more finely divided in the color temperature direction to 
newly generate a plurality of test light sources, and score 
calculation and decisions thereof are performed in stages with 
respect to the newly generated test light sources to further 
improve the resolution of the estimation result. 

When test light sources include a plurality of light 
sources which can be classified in different categories on the 
basis of the physical method of light emission, such as for 
example high-efficiency fluorescent lamps or three-wavelength 
fluorescent lamps, different indexes between evaluations within 
each category and evaluations among categories, are used in 
respective calculations, and score values separately obtained 
may be combined to judge the estimated light source. 

When scene light source estimation processing is performed 
continuously over a period of time, indexes and estimation 
results obtained in the past at short intervals may be combined 
to judge the most recent estimated light source. 

In order to evaluate the correctness of a test light source 
in reflectivity vector space, in addition to the distribution 
state in the P2-P3 plane, distributions in other two-dimensional 
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spaces such as the Pi~p3 and P1-P2 planes may be evaluated, or 
evaluations of one-dimensional distributions along each axis may 
be performed, or the distribution state in three dimensions may 
be evaluated. 

For example, by using a two-dimensional space based on 
relative values among vector channels, such as for example the 
P2/P1-P3/P1 plane resulting from division of both p 2 and p 3 by pi, 
the effects of scattering in exposure for each scene and 
unevenness in the lighting intensity within the same scene on 
evaluations can be suppressed. 

Further, sensor response values, which are the results of 
image pickup of various scenes actually existing or the results 
of prediction by numerical operations of image pickup of diverse 
virtual scenes, can be projected into evaluation spaces for 
separate scenes through operations which can be colorimetrically 
approximated from the spectral sensitivity characteristics of 
the image-pickup means and the spectral . distribution 
characteristics of the image-pickup light source measured at the 
time of image pickup of each scene, and weighting distribution 
and region information generated from the frequency distribution 
of the projection values may be used as a reference color gamut. 

Needless to say, this invention is not limited to the 
above-described embodiment, but can adopt various other 
configurations as appropriate within the scope of the claims of 
this invention. 
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A light source estimation apparatus of this invention 
estimates the correct image-pickup light source by including: 
storage means for storing, for each test light source, 
parameters for projecting sensor response values into an 
evaluation space not dependent on the image-pickup light source 
by performing, for the sensor response values, operations which 
can be colorimetrically approximated from a plurality of 
different known spectral sensitivity characteristics of the 
image-pickup means and the spectral characteristics of a 
plurality of test light sources assumed in advance; projection 
conversion means for projecting sensor response values into the 
evaluation space not dependent on the image-pickup light source 
using parameters stored in the storage means; and evaluation 
means for evaluating the correctness of a plurality of test 
light sources based on the image distribution state of sample 
values of an image scene projected by the projection conversion 
means. Hence by projecting sampled sensor responses into an 
evaluation space not dependent on the light source through 
operations which can be colorimetrically approximated from the 
spectral sensitivity characteristics of the image-pickup system, 
which are known, and from the spectral characteristics of the 
test light source, there is the advantageous result that the 
reasonableness of each test light source can be evaluated based 
on the state of sample values widely distributed therein. 
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Accordingly, it is sufficient to store with respect to each 
test light source only a matrix or other parameters necessary 
for projection from the sensor space into the evaluation space, 
so that by providing evaluation criteria in a single evaluation 
space, high estimation accuracy can be obtained with low memory 
consumption. 

Further, a light source estimation method of this invention 
correctly estimates the image-pickup light source by performing 
projection of sensor response values into an evaluation space 
not dependent on the image-pickup light source through 
operations which can be colorimetrically approximated from 
spectral sensitivity characteristics of the image-pickup means 
which are known and from the spectral characteristics of the 
assumed test light source, and by evaluating the correctness of 
a plurality of test light sources based on the state of 
distribution of sampled values of the projected scene. Hence 
evaluation is performed using a fixed space not dependent on the 
light source, so that it is sufficient to store information for 
only one reference distribution space as the comparison 
criterion for the correct light source, and evaluation 
processing is simplified, so that the problem of cost increases 
can be resolved. Further, a greater amount of information 
(conditions and data) for referencing as criteria for the 
correct light source can be provided, and so there is the 
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advantageous result that optimization adjustment to improve 
estimation accuracy is also facilitated. 

In the method according to this invention, the most 
appropriate light source is judged from among a plurality of 
test light sources; in methods proposed in the prior art, an 
evaluation criterion is necessary for each light source in order 
to perform evaluations in a space which depends on the light 
source, and because the amount of data used as evaluation 
criteria increases in proportion to the number of patterns of 
the test light sources, either the amount of data of evaluation 
criteria or the number of test light sources must be reduced, so 
estimation accuracy is sacrificed, or accuracy is given priority, 
resulting in increased memory costs. In this invention, 
coefficients for space conversion are provided which require 
only a small amount of memory for each test light source, and 
evaluations are performed using a fixed space not dependent on 
the light source, so that it is sufficient to store with respect 
to only a single space the information (conditions and data) to 
be referenced as comparison criteria for judging the correct 
light source, and consequently the estimation accuracy can be 
improved without increases in cost, affording advantages over 
the techniques of the prior art. 

Further, an image-pickup apparatus of this invention 
includes: storage means for storing, for each test light source, 
parameters for projecting sensor response values into an 



evaluation space not dependent on the image-pickup light source 
by performing operations which can be colorimetrically 
approximated, for the sensor response values, from a plurality 
of different known spectral sensitivity characteristics of the 
image-pickup means and the spectral characteristics of a 
plurality of test light sources assumed in advance; projection 
conversion means for projecting sensor response values into the 
evaluation space not dependent on the image-pickup light source 
using parameters stored in the storage means; evaluation means 
for estimating the correct image-pickup light source by 
evaluating the correctness of a plurality of test light sources 
based on the image distribution state of sample values of an 
image scene projected by the projection conversion means; light 
source estimation means for estimating the final image-pickup 
light source to be determined as the estimated light source by 
conjoining in numerical formulas, or by selecting through 
conditional branching, or by combining both of, an image-pickup 
light source determined by estimation and a light source 
determined by an estimation method different from the estimation 
method used; and color balance adjustment means for using 
spectral characteristics, which are the color of the estimated 
image-pickup light source, or parameters appropriate thereto in 
color balance processing of the sensor response of the image- 
pickup means. Hence in the image-pickup apparatus, the range of 
estimation of the image-pickup light source can be broadened, 
and it is only necessary to store a matrix or other parameters 



for each test light source for projection from sensor space into 
evaluation space, and so there is the advantageous result that 
by providing evaluation criteria in a single evaluation space, 
high estimation accuracy can be obtained with low memory 
consumption, enabling use in color balance processing. 

Further, an image processing method of this invention 
performs projection, for sensor response " values, into an 
evaluation space not dependent on the image-pickup light source 
through operations which can be colorimetrically approximated 
from known spectral sensitivity characteristics of the image- 
pickup means and from the spectral characteristics of the 
assumed test light source; estimates the correct image-pickup 
light source by evaluating the correctness of a plurality of 
test light sources based on the distribution state of sample 
values of the projected scene; estimates the final image-pickup 
light source to be determined as the estimated light source by 
conjoining in numerical formulas, or by • selecting through 
conditional branching, or by combining both of, an image-pickup 
light source determined by estimation and a light source 
determined by an estimation method different from the estimation 
method used; and uses the spectral characteristics, which are 
the color of the estimated image-pickup light source, or 
parameters appropriate thereto in color balance processing of 
the sensor response of the image-pickup means. Hence in the 
image processing method, the range of estimation of the image- 



pickup light source can be broadened, and it is only necessary 
to store a matrix or other parameters for each test light source 
for projection from sensor space into evaluation space, and so 
there is the advantageous result that by providing evaluation 
criteria in a single evaluation space, high estimation accuracy 
can be obtained with low memory consumption, enabling use in 
color balance processing. 

This invention can provide one framework for accurately 
estimating the light source of an image-pickup scene from the 
response of the image-pickup system. If the light source of an 
unknown scene can be estimated in the image-pickup system, it 
becomes possible to accurately determine the image white balance 
adjustment, color matching adjustment and other parameters in 
the image-pickup equipment, and accurate color reproduction of 
the picked up image of a scene, and accurate correction so as to 
obtain a specific intended color reproduction, can be performed 
when recording and displaying images. 



36 



